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Abstract 

In this paper we present two Douglas-Rachford inspired iteration schemes which can be applied 
directly to JV-set convex feasibility problems in Hilbert space. Our main results are weak convergence 
of the methods to a point whose nearest point projections onto each of the N sets coincide. For 
afhne subspaces, convergence is in norm. Initial results from numerical experiments, comparing our 
methods to the classical (product-space) Douglas-Rachford scheme, are promising. 

1 Introduction 

Given N closed and convex sets with nonempty intersection, the N-set convex feasibility problem asks 
for a point contained in the intersection of the TV sets. Many optimization and reconstruction problems 
can be cast in this framework, either directly or as a suitable relaxation if a desired bound on the quality 
of the solution is known a priori. 

A common approach to solving iV-set convex feasibility problems is the use of projection algorithms. 
These iterative methods assume that the projections onto each of the individual sets are relatively simple 
to compute. Some well known projection methods include von Neumann's alternating projection method 
[38, 26, 17, 3, 6, 28, 29, 34], the Douglas-Rachford method [20, 31, 10] and Dykstra's method [22, 16, 4]. 
Of course, there are many variants. For a review, we refer the reader to any of [2, 5, 19, 37, 24, 13]. 

On certain classes of problems, various projection methods coincide with each other, and with other 
known techniques. For example, if the sets are closed affine subspaces, alternating projections = Dykstra's 
method [16]. If the sets are hyperplanes, alternating projections = Dykstra's method = Kaczmarz's 
method [19]. If the sets are half-spaces, alternating projections = the method Agmon, Motzkin and 
Schoenberg (MAMS), and Dykstra's method = Hildreth's method [24, Chapter 4]. Applied to the phase 
retrieval problem, alternating projections = error reduction, Dykstra's method = Ficnup's BIO, and 
Douglas-Rachford = Ficnup's HIO [8]. 

Continued interest in the Douglas-Rachford iteration is in part due to its excellent — if still largely 
mysterious — performance on various problems involving one or more non-convex sets. For example, in 
phase retrieval problems arising in the context of image reconstruction [8, 9]. The method has also been 
successfully applied to NP-complete combinatorial problems including Boolean satisfiability [23, 25] and 
Sudoku [23, 36]. In contrast, von Neumann's alternating projection method applied to such problems 
often fails to converge satisfactorily. For progress on the behaviour of non-convex alternating projections, 
we refer the reader to [30, 11, 27, 21]. 

Recently, Borwein and Sims [14] provided limited theoretical justification for non-convex Douglas- 
Rachford iterations, proving local convergence for a prototypical Euclidean case involving a sphere and 
an affine subspace. For the two-dimensional case of a circle and a line, Borwein and Aragon [1] were 
able to give an explicit region of convergence. Even more recently, a local version of firm nonexpansivity 
has been utilized by Hesse and Luke [27] to obtain local convergence of the Douglas-Rachford method 
in limited non-convex settings. Their results do not directly overlap with the work of Aragon, Borwein 
and Sims (for details see [27, Example 43]). 

Most projection algorithms can be extended in various natural ways to the iV-set convex feasibility 
problem without significant modification. An exception is the Douglas-Rachford method, for which only 
the theory of 2-set feasibility problems has so far been successfully investigated. For applications involv- 
ing N > 2 sets, an equivalent 2-set feasibility problem can, however, be posed in a product space. We 
shall revisit this later in our paper. 



The aim of this paper is to introduce and study the cyclic Douglas-Rachford and averaged Douglas- 
Rachford iteration schemes. Both can be applied directly to the N-set convex feasibility problem without 
recourse to a product space formulation. 

The paper is organized as follows: In Section 2, we give definitions and preliminaries. In Section 3, 
we introduce the cyclic and averaged Douglas-Rachford iteration schemes, proving in each case weak 
convergence to a point whose projections onto each of the constraint sets coincide. In Section 4, we 
consider the important special case when the constraint sets are affine. In Section 5, the new cyclic 
Douglas-Rachford scheme is compared, numerically, to the classical (product-space) Douglas-Rachford 
scheme on feasibility problems having ball or sphere constraints. Initial numerical results for the cyclic 
Douglas-Rachford scheme are quite positive. 

2 Preliminaries 

Throughout this paper, 



H is a real Hilbert space with inner product (■, ■) 



and induced norm || • ||. We use — 1 to denote weak convergence. 
We consider the N-set convex feasibility problem: 



N 



Find x G P| Ci ^ where d C H are closed and convex. (1) 



i=l 



Given a set S C % and point x E H, the best approximation to x from S is a point p € S such that 

\\p — x\\ = d(x, S) :— inf \\x — s\\ . 

If for every x E T-L there exists such a p, then S is said to be proximal. Additionally, if p is always unique 
then S is said to be Chebyshev. In the latter case, the projection onto S is the operator P$ : H — > S 
which maps x to its unique nearest point in S and we write Ps{x) = p. The reflection about S is the 
operator Rg : % — > % defined by R$ '■= 2Pg — / where I denotes the identity operator which maps any 
x € % to itself. 

Fact 2.1. Let C C "H be non-empty closed and convex. Then: 
(i) C is Chebyshev. 
(ii) ( Characterization of projections ) 

Pc{x) = p <^=> (x — p, c — p) < for all c 6 C. 

(Hi) ( Characterization of reflections ) 

Rc (x) = r (x — r,c — r) < —\\x— r\\ 2 for all c E C. 

(iv) (Translation formula) For y EH, P y+ c( x ) = V + Pc{% ~ y)- 
(v) (Dilation formula) For O^Ael, P\c(x) = XP c {x/X). 
(vi) If C is a subspace then Pq is linear. 
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(vii) If C is an affine subspace then Pq is affine. 

Proof. See, for example, [7, Theorem 3.14, Proposition 3.17, Corollary 3.20], [24, Theorem 2.8, Exer- 
cise 5.2(i), Theorem 3.1, Exercise 5.10] or [37, Theorem 2.1.3, Theorem 2.1.6]. Note, the equivalence of 
(ii) and (iii) by substituting r — 2p — x. □ 

Given A, B C % we define the 2-set Douglas- Rachford operator Ta.b '■ "H. —> H by 

I + R b Ra ,„s 

J AB •= 2 ' ' ' 

Note that Ta,b and Tb,a & re typically distinct, while for an affine set A we have Ta.a = I ■ 

The basic Douglas-Rachford algorithm originates in [20] and convergence was proven as part of [31]. 

Theorem 2.1 (Douglas-Rachford [20], Lions-Mercier [31]). Let A,BCH be closed and convex with 
nonempty intersection. For any Xq G W, the sequence T\ b Xq converges weakly to a point x such that 
P A x G An B. 

Theorem 2.1 gives an iterative algorithm for solving 2-set convex feasibility problems. For applications 
involving N > 2 sets, an equivalent 2-set formulation is posed in the product space H . This is discussed 
in detail in Remark 3.4. 

Let T : H — > %. We recall that T is asymptotically regular if T n x — T n+1 x — > 0, in norm, for all 
x G U. We denote the set of fixed points of T by FixT = {x : Tx = x}. Let D C U and T : D -> U. We 
say T is nonexpansive if 

\\Tx - Ty\\ < \\x - y\\ for all x,y G D 
(i.e. 1-Lipschitz). We say T is firmly nonexpansive if 

\\Tx-Tyf + \\(J-T)x- {I-T)y\\ 2 < \\x - y\\ 2 for all x,y e D. 

It immediately follows that every firmly nonexpansive mapping is nonexpansive. 

Fact 2.2. Let A, B C H be closed and convex. Then Pa is firmly nonexpansive, Ra is nonexpansive and 
Ta,b is firmly nonexpansive. 

Proof. See, for example, [7, Proposition 4.8, Corollary 4.10, Remark 4.24], or [37, Theorem 2.2.4, Corol- 
lary 4.3.6]. □ 

The class of nonexpansive mappings is closed under convex combinations, compositions, etc. The class 
of firmly nonexpansive mappings is, however, not so well behaved. For example, even the composition 
of two projections onto subspaces need not be firmly nonexpansive (see [6, Example 4.2.5]). 

A sufficient condition for firmly nonexpansive operators to be asymptotically regular is the following. 

Lemma 2.1. Let T : % — > % be firmly nonexpansive with FixT ^ 0. Then T is asymptotically regular. 

Proof. See, for example, [35, Corollary 1] or [37, Lemma 4.3.5]. □ 

The composition of firmly nonexpansive operators is always nonexpansive. However, nonexpansive 
operators need not be asymptotically regular. For example, reflection with respect to a singleton, clearly 
is not; nor are most rotations. The following is a sufficient condition for asymptotic regularity. 

Lemma 2.2. Let Ti : H —> H be firmly nonexpansive, for each i, and define T := T r . . . T% T\. If 
Fix T ^ then T is asymptotically regular. 

Proof. See, for example, [7, Theorem 5.22]. □ 

Remark 2.1. Recently Bauschke, Martm-Marquez, Moffat and Wang [12, Theorem 4.6] showed that any 
composition of firmly nonexpansive, asymptotically regular operators is also asymptotically regular, even 
when FixT = 0. 

The follow lemma characterizes fixed points of certain compositions of firmly nonexpansive operators. 
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Lemma 2.3. Let Ti : H —> H be firmly nonexpansive, for each i, and define T := T r . . . T% T\. If 
n[ =1 Fix T t ? then Fix T = f|[ =1 Fix T t . 

Proof. See, for example, [7, Corollary 4.37]. □ 

There are many way to prove Theorem 2.1. One is to use the following well-known theorem of Opial 
[33]. 

Theorem 2.2 (Opial). Let T : % — > H be nonexpansive, asymptotically regular, and FixT =/= 0. Then 
for any xo G %, T n xo converges weakly to an element of FhiT. 

Proof. See also, for example, [33] or [7, Theorem 5.13]. □ 
In addition, when T is linear, the limit can be identified and convergence is in norm. 

Theorem 2.3. LetT -.'H—t'H be linear, nonexpansive and asymptotically regular. Then for any xq G H, 

in norm, 

lim T n x = Pf-^tXq. 

n— >oc 

Proof. See, for example, [7, Proposition 5.27]. □ 

Remark 2.2. A version of Theorem 2.3 was used by Halperin [26] to show that von Neumann's alter- 
nating projection, applied to finitely many closed subspaces, converges in norm to the projection on the 
intersection of the subspaces. 1 

Summarizing, we have the following. 

Corollary 2.1. Let Tj : H — > H be firmly nonexpansive, for each i, with P|^ =1 Fix7i ^ and define 
T := T r . . . TiT\ . Then for any xo G %, T n xo converges weakly to an element of FixT = C\f =1 FixTj. 
Moreover, ifT is linear, then T n xo converges, in norm, to PfixtXq- 

Proof. Since T is the composition of nonexpansive operators, T is nonexpansive. By Lemma 2.3, FixT =/= 
0. By Lemma 2.2, T is asymptotically regular. The result now follows by Theorem 2.2 and Theorem 2.3. 

□ 

We note that the verification of many results in this section can be significantly simplified for the 
special cases we require. 

3 Cyclic Douglas— Rachford Iterations 

We are now ready to introduce our first new projection algorithm, the cyclic Douglas- Rachford iteration 
scheme. Let C±, C 2 , . . . , Cn C % and define Tj Cl c 2 ... Cjv ] : % — >• H by 

[Ci C*2 •■ ■ Cn] ' C n 1C1 Cn -i ,C ' n ■ * * Ci^C^, C±,C2 

I + R Cl Rc N ^ ri + Rc N Rc N .A n + Rc 3 Rc 2 \ ^ I + Rc 2 R Cl 
Given xq G W, the cyclic Douglas- Rachford method iterates by repeatedly setting 

X n +1 — T[ Cl C2 C N ] x n- 

Remark 3.1. In the two set case, the cyclic Douglas-Rachford operator becomes 

rr, rp rp { I + RClRC 3 \ { I + RCl^Cl 

1 [c 1 c 2 ] - J-c 2 ,c 1 -Lc 1 ,c 2 - I 2 I { 2 

That is, it does not coincide with the classic Douglas-Rachford scheme. 



1 Kakutani had earlier proven weak convergence for finitely many subspaces [32]. Von Neumann's original two-set proof 
does not seem to generalize. 
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Where there is no ambiguity, we take indices modulo N, and abbreviate Tc u Cj by Tjj, and 
T[c 1 c 2 ...c N ] by T [12 ...jv]- In particular, T 0<1 := T NA , T N<N+1 := T N;1 , C := C N and C N +i := C\. 



Recall the following characterization of fixed points of the Douglas-Rachford operator. 
Lemma 3.1. Let A,B t-H. be closed and convex with nonempty intersection. Then 

Pa Fix Ta,b = AnB. 

Proof. P A FixT AyB CAnB since 

x e FixT A .B <s=> x + r b RaX = x ^ p AX = P b Rax eAnB. 

It is straightforward to check the reverse inclusion. □ 

We are now ready to present our main result regarding convergence of the cyclic Douglas-Rachford 
scheme. 

Theorem 3.1 (Cyclic Douglas-Rachford). Let C\, C2, . . . , Cjv ^ H be closed and convex sets with a 
nonempty intersection. For any xq G %, the sequence Tj™ 2 n ^xq converges weakly to a point x such that 

PdX = Pcj x j f or all indices i,j. Moreover, PcjX € Di=i Q> f or eac ^ index j. 
Proof. By Fact 2.2, T iii+ i is firmly nonexpansive, for each i. Further, 

N N 



f)FixT M+1 d f]a ? 



By Corollary 2.1, T^ 2 N j%a converges weakly to a point x g FixT[ 12 ...jv] = n^i^ x ^i,j+i- By 
Lemma 3.1, Pd x € Ci+i, for each i. Now we compute 



N , N 



\ E W P ^ x - P c*-A\ 2 = M) + i £ (W P cA\ 2 - 2<Pc<x, Pc^x) + \\Pc^x\\ 



2^" ~* 11 4 ' ' 2 

i=l i=l 



IN \ N N 

= i x, ^(p Cj _^ - p Ci x) ) - ]T(Pc^, Pc.^x) + x: n^xf 

\ i=l / i=l i=l 

AT 

Fact 2.1 



1=1 

Thus, PdX — Pc i _ 1 x, for each i; and we are done. □ 

Again by invoking Opial's Theorem, a more general version of Theorem 3.1 can be abstracted. 

Theorem 3.2. Let Ci, C2, ■ ■ • , Cn C % be closed and convex sets with nonempty intersection, let Tj : 
H — > H, for each j, and define T := T/v . . .T%Tx. Suppose the following three properties hold. 

1. T = Tm . . . X2T1, is nonexpansive and asymptotically regular, 

2. FixT = n*LiFixTj + 0, 

3. Pcj FixTj C Cj+i, for each j . 

Then, for any xq G H, the sequence T n xo converges weakly to a point x such that Pqx = PcjX for all 
i,j. Moreover, PcjX € (")i=a Ci, for each j. 

Proof. By Theorem 2.2, T n xo converges weakly to point x € FixT. The remainder of the proof is the 
same as Theorem 3.1. □ 
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Remark 3.2. We give a sample of examples of operators which satisfy the three conditions of Theorem 3.2. 

1- T{A X a 2 ...a m ] wnere Aj G {C\,C 2 ...Cjv}, and is such that each Cj appear in the sequence 
Ax, A2, . ■ ■ , Am at least once. 

2. T is any composition of Pc lt Pc 2 i ■ ■ ■ > -Pcnj such that each projection appears in said composition 
at least once. In particular, setting T — Pc N ■ ■ ■ Pc 2 Pc x we recover Bregman's seminal result [17]. 

3. Tj = (I + Pj)/2 where Pj is any composition of Pc 1 ,Pc 2 i ■ ■ ■ ^Pc N sucn that, for each i, there 
exists a j such that Pj = PdQj for some composition of projections Qj. A special case is, 

T= ( I + Pc x Pc N \ [ I + P Cs Pc^ ^ I + Pc 2 P Cl 

4. If Ti, T 2 . . . , Tjv/ are operators satisfying the conditions of Theorem 3.2, replacing Tj with the relax- 
ation ctjl + (1 — ctj)Tj where a,j G]0, 1/2], for each i. Note the relaxations are firmly nonexpansive 
[7, Remark 4.27]. 

Of course, there are many other applicable variants. For instance, Krasnoselski-Mann iterations (see 
[7, Theorem 5.14] and [15]). 

We now investigate the cyclic Douglas-Rachford iteration in the special-but-common case where the 
initial point lies in one of the target sets; most especially the first target set. 

Corollary 3.1. Let C\, C 2 , . . . , Cjq C ft be closed and convex sets with a nonempty intersection. If 
y G Ci then Ti^ + \y = Pc i+1 y. In particular, if xo G C\, the cyclic Douglas-Rachford trajectory coincides 
with that of von Neumann's alternating projection method. 

Proof. For any y € H, T iti+1 y = P Cl+l y Rc, +1 V = Rc i+l RciV- H V € Ci then R c ,y = y. In 

particular, if xq G C\ then 

P\\2...n\Xq = T N<1 . . . T 2 ^T 12 y = Pc^Cn ■ --Pc^o G Ci, 

and the result follows. □ 

Remark 3.3. If xq ^ C\, then the cyclic Douglas-Rachford trajectory need not coincide with von Neu- 
mann's alternating projection method. We give an example involving two closed subspaces with codi- 
mension 1 (see Figure 1). Define 

Ci := {x G H : (a u x) = 0}, C 2 :={i€H: (a 2 ,x) = 0}, 

where ai,a 2 G H such that (ai,a 2 ) 7^ 0. By scaling if necessary, we may assume that ||ai|| = 1 1 a. 2 1 1 = 1- 
Then one has, 

Pc^x = x — (ai,x)ai, Pc 2 x = x — (a 2 ,x)a 2 , 



and 



Similarly, 



By Remark 4.1, 



T 1>2 x = x + 2P C2 P Cl x - (P Cl x + P C2 x) 

= x — {ax,x)ax - (a 2 ,x)a 2 + 2(ai,a 2 )(ai,x)a 2 . 



T 2 ,\x = x — {ai,x)ai — (a 2 ,x)a 2 + 2(ai,a 2 )(a 2 ,x)ai. 

2(ai,T[ 12 ] x ) = (ax,T lt2 x) + (a 1 ,T 2; ix) 

= (ai,x) - (ai,.T)||ai|| 2 - (a 2 ,x)(a 1 ,a 2 ) 

+ (ai,a 2 )(a 2 ,x)\\a 1 \\ 2 + (a 1 ,a 2 ) 2 (a 1 ,x) 
= (a 1 ,a 2 ) 2 (ax,x). 







Figure 1: An interactive Cinderella applet showing a cyclic Douglas-Rachford trajectory differing from 
von Neumann's alternating projection method. Each green dot represents a 2-set Douglas-Rachford 
iteration. 




Figure 2: An interactive Cinderella applet showing a cyclic Douglas-Rachford trajectory differing from 
von Neumann's alternating projection method. Each green dot represents a 2-set Douglas-Rachford 
iteration. 
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Similarly, 2(a 2 ,T [12 ] x ) = (ai, a 2 ) 2 (a 2 , x). 

Thus, if (ai,x} 0, for each i, then (ai,Tu2]x) ^ 0, for each i. In particular, if xo ^ Ci U C2, then 
none of the cyclic Douglas-Rachford iterates lie in C\ or C 2 . 

A second example, involving a ball and an affine subspace is illustrated in Figure 2. 

Remark 3.4 (A product version). We now consider the classical product formulation of (1). Define two 
subsets of H N : 

N 

C:=Y[d, D :={(x,x,...,x) eU N :xeU}, (3) 

i=l 

which are both closed and convex (in fact, D is a subspace). Consider the 2-set convex feasibility problem 

FindxeCnDCH* (4) 
Then (1) is equivalent to (4) in the sense that 

JV 

x e P| c s ; (x,x,...,x) eCnD. 
»=i 

Further the projections, and hence reflections, are easily computed since 



JV JV I JV \ 

i=l »=1 \ J=l / 



Let Xo € .D and define x„ := T[£> <7]X rl _i. Then Corollary 3.1 yields 
%>C]X« = PdPc^u = 



/ JV JV JV 

\ i=l i=l t=l 



That is, if — as is reasonable — we start in D, the cyclic Douglas-Rachford method coincides with averaged 
projections. 

In general, the iteration is based on 

T [DC] x = x - P D x + 2P D P c T DtC x - P C T D ,c* + PcRd* - P d P C Rd*- (5) 
If x = [x\, X2, ■ ■ ■ , Xn), then the ith coordinate of (5) can be expressed as 

JV „ JV 



(T [D c] x) 4 = Xi - ± x j + TJ 12 P c t ( T n,c*)i - Pc, (T D ,cx)» 



J N ■ 
j=i 3=1 



2 



1 '< I jf Y, x i '- Xi J - jj E (I? E - ■'■■< 1 



where 



1 W / 2 W \ 

(T AC x)^ Xk + P 3* hv E Xfe - ^ ' 

k=l \ k=l J 



which is a considerably more complex formula. 
Let A, B C %. Recall that points (x, y) € A x B form a best approximation pair relative to (A,B) if 
||x - y\\ = d(A, B) := inf{||a - b\\ :aeA,be B}. 
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Remark 3.5. (a) Consider C\ = B-u := {.i £ W : ||x|| < 1} and C2 = {y}, for some y £ T~L. Then 

T[i2]X = x- P Cl x + P Cl (y - x + P Cl x), 
where Pc x z = z if z G C\, and ^/||^|| otherwise. Now, 

xeFixT [12 ] P Cl x = P Cl {y - x + P Cl x). (6) 

Thus, 

• If x € C\ then x = Pc^y- 

• Ify — x + Pc ± x G C\ then x = y. 

• Else, ||x|| > 1 and \\y - x + P A x\\ > 1. By (6), 

x = Xy where A = ( T . -Mj ^ ) e]0, 1[. 

\\\y - x + P Cl x\\ + \\x\\ - 1 J 

Moreover, since 1 < ||x|| = A||y||, we obtain A e]l/||j/||, 1[. 

In each case, Pc x x = Pc ± y and Pc 2 x = y. Therefore {Pc 1 x,Pc 2 x) is a best approximation pair relative 
to (Ci, C2) (see Figure 3). In particular, if C\ n C2 7^ 0, then P^J/ = y and, by Theorem 3.1, the cyclic 
Douglas-Rachford scheme weakly converges to y, the unique element of C\ n C*2- 

When Ci H C2 = 0, Theorem 3.1 cannot be invoked to guarantee convergence. However, the above 
analysis provides the information that 

FixT [12] C{AP Cl y+(l-%:Ae [0,1]}- 

(b) Suppose instead, C± = Su := {x G H : ||x|| = 1}. A similar analysis can be performed. If y ^ and 
x S FixTr 12 i are such that x, y — x + Pc\x 7^ 0, then 

• If x € C\ then x = Pc x y- 

• If y — x + Pc x x E C\ then x = y. 

• Else, x — Xy where 



A {\\y-x + P Cl x\\ + \\x\\-l) ~ ^llp-a-IH-llP^H + llxll-lJ > °' 

Again, (Pc\x, Pc 2 x) is a best approximation pair relative to (G\,C2). 

Experiments with interactive Cinderella 2 dynamic geometry applets, suggest similar behaviour of the 
cyclic Douglas-Rachford method applied to many other problems for which C\ n C2 = 0. For example, 
see Figure 4. This suggests the following conjecture. 

Conjecture 3.1. Let C\, C2 C % be closed and convex with CiHC^ = 0. Suppose that a best approxima- 
tion pair relative to [C\,C2) exists. Then the two-set cyclic Douglas-Rachford scheme converges weakly 
to a point x such that (Pc 1 x, Pc 2 x ) is a best approximation pair relative to the sets (C±,C2)- 

Remark 3.6. If there exists an integer n such that either T^^xg g C\ or Ti^T^^xq € C2, by Corol- 
lary 3.1, the cyclic Douglas-Rachford scheme coincides with von Neumann's alternating projection 
method. In this case, Conjecture 3.1 holds by [18, Theorem 2]. In this connection, we also refer the 
reader to [3, 4]. 

It is not hard to think of non-convex settings in which Conjecture 3.1 is false. For example, in R, let 
C\ = [0, 1] and C2 = {0, ^}. If xo = 1 then Tji2]Xo = xq, but 

(P Cl (l),P C2 (l)) = (l,^), 
which is not a best approximation pair relative to (Ci,C2). 
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Figure 3: An interactive Cinderella applet showing the behaviour described in Remark 3.5. Each green 
dot represents a cyclic Douglas-Rachford iteration. 
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Figure 4: An interactive Cinderella applet showing the cyclic Douglas-Rachford method applied to the 
case of a non-intersecting ball and a line. The method appears convergent to a point whose projections 
onto the constraint sets form a best approximation pair. Each green dot represents a cyclic Douglas- 
Rachford iteration. 
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We now present an averaged version of our cyclic Douglas-Rachford iteration. 

Theorem 3.3 (Averaged Douglas-Rachford). Let C\, C2, . . . , Cjy C % be closed and convex sets with a 
nonempty intersection. For any Xq G %, the sequence defined by 



converges weakly to a point x such that PdX = PcjX for all indices i,j. Moreover, Pc } x G Pli=i f or 
each index j . 

Proof. Consider C,DCH N as (3) and define T := Pd(U^=i B Y Fact 2.2, P D is firmly nonexpan- 

sive. By Fact 2.2, X^i+i is firmly nonexpansive in H, for each i, hence Yli=i i s firmly nonexpansive 

in H N . Further, Fix(rj. =1 T^i+i) n P& 2 C <~) D 7^ 0. By Corollary 2.1, x„ converges weakly to a point 
xeFixT. 

Let Xo = (xo, Xo, . . . , xq) G H N . Since Tx„ G D, for each n, we write x„ = (x n ,x n , . . . , x n ) for some 
x„ G H. Then 



Y j m+i I 

i=l / 



independent of i. Similarly, since x G FixPo = D, we write x = (x,x, . . . ,x) G % for some x G 
Since x G Fix(FJ i=1 Tj^+i), x G FixTj^+i, for each i, and hence PdX G Cj+i. The same computation as 
in Theorem 3.1 now completes the proof. □ 

Since each 2-set Douglas-Rachford iteration can be computed independently, the averaged iteration 
is easily parallelizable. 

4 Affine Constraints 

In this section we observe that the conclusions of Theorems 3.1 and 3.3 can be strengthened when the 
constraints are affinc. 

Lemma 4.1 (Translation formula). Let C[,C' 2 , . . . , C' N CM be closed and convex sets with a nonempty 
intersection. For fixed y G H, define Ci := y + C[, for each i. Then 

T Cu c l+1 x = y + T C '.,c i+1 (x - y), 

and 

T [Cl c 2 ...c N ]X = y + T [C f/ cf /... c ^ ] (x-y). 
Proof. By the translation formula for projections (Fact 2.1), we have 

RdX = y 4- Rc" {x — y), for each i. 

The first result follows since, 

x + R Ci+1 R c .x x + Rc i+1 (y + RcXx - y)) 



Tci,c i+1 x 



2 2 

x + y + R c - Rc> (x -y) (x-y) + Rc> Rc[ (x - y) 
=y + Jr 1 — * 



2 

= y + Tc i ,c' i+1 {x-y). 

Iterating gives, 

Tc 2 ,c 3 T Cu c 2 = T C2 ,c 3 (y + Tc^c^x - yj) = y + T c ^ c >T c ^ ai (x - y), 
from which the second result follows. □ 



"See http://www.cinderella.de/. 
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Theorem 4.1 (Norm convergence). Let C\, C2, • • • , Cn QH be closed affine subspaces with a nonempty 
intersection. Then, for any xq £ T-L, 

£^ T \PxCz...c N ] x o = p FixT [Cl O2 ... cjvl a:o, 

is norm convergent. 

Proof. Let c € nf^CV Since Cj are affine we write Ci = c + C-, where C[ is a closed subspace. Since 
Tc",c' +1 is linear, for each i, so is Tjcj c 2 c^]- By Fact 2.2, for each i, Tc'.,c'. +1 is firmly nonexpansive. 
Further, D? =1 FixT C / jC / j D H^CI 7^ 0. By Lemma 4.1 and Corollary 2.1, 

T [ Cl c 2 ... c N ]Z = c + T [c[ c> 2 ... C N ] {x~c)^c + P Fix T [c , c , ... CJv , (x - c) = P F ixT [Cl C2 ... Cn] x. 
This completes the proof. □ 
Remark 4.1. For the case of two closed affine subspaces, the iteration becomes 
m rri rri I + Pb Ra + RaRb + RaRb Rb Ra 2I + RbRa + RaRb Ta.b+Tb.a 

J-[AB] - IbMa.B ~ 1 - ^ - 2 ■ 

That is, the cyclic Douglas-Rachford and averaged Douglas-Rachford methods coincide. 

For N > 2 closed affine subspaces, the two methods do not always coincide. For instance, when 
N = 3, 

^[12 3] = ^3,1^2, 3^1, 2 

= I - {P Cl + Pc 2 + Pc 3 ) + (P Cl Pc 3 + Pc 2 P Cl + Pc 3 Pc 2 + Pc 3 P Cl + P Cl Pc 2 ) 
- (Pc 3 Pc 2 P Cl + P Cl Pc 3 Pc 2 + P Cl Pc 3 P Cl + P Cl Pc 2 P Cl ) + 2P Cl Pc 3 Pc 2 P Cl , 

which includes a term which is the composition of four projection operators. 

Theorem 4.2 (Averaged norm convergence). Let C\, C2, • • • , Cn Q W be closed affine subspaces with a 
nonempty intersection. Then, in norm 



'[Ci c 2 ...o N ) 



x . 



Proof. Let C, D C H N as in (3). Let c g C\f =l Ci and define c = (c, c, . . . , c) £ H . Since Ci are affine 
we may write Ci = c + C|, where (7,- is a closed subspace, and hence C = c + C where C — niLi 

For convenience, let Q denote n»=i Tc^C! t an d let T = PdQ- Since C and D are subspaces, T is 
linear. By Fact 2.2, Tc',C +1 is firmly nonexpansive, hence so is Q. Further, FixT D FixQ fl Fix Pd 2 
FixQ n D ^ since n^C- ^ 0. 

As a consequence of Lemma 4.1, we have the translation formula 

Tx = c + T(x - c), for any x e H N . 

As in the proof of Theorem 4.1, the translation formula, together with Corollary 2.1, shows T"x — s- 
PkerT^o =: z where xo = (xq, Xq, . . . , xq) G H N . As x„ £ D, we may write x„ = (x n , x n , . . . , x n ) for 
some x„ G %. Similarly, we write z = (z, z, . . . , z). Then 

\/N\\x - z\\ = ||x - z|| = d(x , FixT) 

< d(xo, (nji 1 FaT iii+ i) JV ) = VJV^o.n^ FixT i;i+1 ). 

Hence, z = P n « FixTii+i^o- By Lemma 2.3, FixT^ q 2 c n ] — ^tL\ Pix^i.i+lj an d the proof is com- 
plete. □ 
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Figure 5: An interactive Cinderella applet using the cyclic Douglas-Rachford method to solve a feasibility 
problem with two sphere constraints. Each green dot represents a 2-set Douglas-Rachford iteration. 



5 Numerical Experiments 

In this section we present the results of computational experiments comparing the cyclic Douglas- 
Rachford and (product-space) Douglas-Rachford schemes — as serial algorithms. These are not intended 
to be a complete computational study, but simply a first demonstration of viability of the method. From 
that vantage-point, our initial results are promising. 

Two classes of feasibility problems were considered, the first convex and the second non-convex. 

N 

(PI) Find x € p| Q C R n where C, = Xi + nB n := {y : \\x t - y\\ < rj, 

i=i 

N 

(P2) Find x £ P| C t C R" where C, = x t + nS n := {y :\\ Xi - y\\ = n}. 

i=i 

Here B-h (resp. S-u) denotes the closed unit ball (resp. unit sphere). 

To ensure all problem instances were feasible, constraint sets were randomly generated using the 
following criteria. 

• Ball constraints: Randomly choose Xi € [—5,5]" and r< € [|| 1 1 , +0.1]. 

• Sphere constraints: Randomly choose Xi £ [—5,5]™ and set r t = \\xi\\. 

In each cases, by design, the non-empty intersection contains the origin. We consider both over- and 
under-constrained instances. 

Note, if Ci is a sphere constraint then Pc i (xi) = Ci (i.e., nearest points are not unique), and Pc t a 
set- valued mapping. In this situation, a random nearest point was chosen from Ci. In every other case, 
Pd is single valued. 

For the comparison, the classical Douglas-Rachford scheme was applied to the equivalent feasibility 
problem (4), which is formulated in the product space (M. n ) N . 

Computations were performed using Python 2.6.6 on an Intel Xeon E5440 at 2.83GHz (single threaded) 
running 64-bit Red Hat Enterprise Linux 6.4. The following conditions were used. 

• Choose a random xo £ [—10, 10]™. Initialize the cyclic Douglas-Rachford scheme with xq, and the 
parallel Douglas-Rachford scheme with (xo, xq, . . . , xo) € (R n ) . 

• Iterate by setting 

x n+ i = Tx n where T = T[ 12 ...jv] or T c .d- 
An iteration limit of 1000 was enforced. 
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Figure 6: Cyclic Douglas-Rachford algorithm applied to a 3-set feasibility problem in IR 2 . The constraint 
sets are colored in blue, red and yellow. Each arrow represents a 2-set Douglas-Rachford iteration. 



3.7 




1.3 



Figure 7: Cyclic Douglas-Rachford algorithm applied to a 3-set feasibility problem in M 3 . The constraint 
sets are colored in blue, red and yellow. Each arrow represents a 2-set Douglas-Rachford iteration. 



• Stopping criterion: 

\\x n - x n+ i\\ < e where e = 1CT 3 or 10~ 6 . 

• After termination, the quality of the solution was measured by 

TV 

error = H-Pci^ — Pci x \\ 2 ■ 

Results are tabulated in Tables 1, 2, 3 & 4. A "0" error (without decimal place) represents zero 
within the accuracy the numpy . float 64 data type. Illustrations of low dimensional examples are shown 
in Figures 5, 6 and 7. 

We make some comments on the results. 

• The cyclic Douglas-Rachford method easily solves both problems. 

Solutions for 1000 dimensional instances, with varying numbers of constraints, could be obtained in 
under half-a-second, with worst case errors in the order of 10 -13 . Many instances of the (PI) where 
solved without error. Instances involving fewer constraints required a greater number of iterations 
before termination. This can be explained by noting that each application of Tu2...N] applies a 



14 



2-set Douglas-Rachford operator TV times, and hence iterations for instances with a greater number 
of constraints are more computationally expensive. 

When the number of constraints was small, relative to the dimension of the problem, the Douglas- 
Rachford method was able to solve (PI) in a comparable time to the cyclic Douglas-Rachford 
method. 

For larger numbers of constraints the method required significantly more time. This is a conse- 
quence of working in the product space, and would be ameliorated in a parallel implementation. 

Applied to (P2), the original Douglas-Rachford method encountered difficulties. 

While it was able to solve (P2) reliably when e = 10 -3 , when e = 10 -6 the method failed to 
terminate in every instance. However, in these cases the final iterate still yielded a point having a 
satisfactory error. The number of iterations and time required, for the Douglas-Rachford method 
was significantly higher compared to the cyclic Douglas-Rachford method. As with (PI), the 
difference was most noticeable for problems with greater numbers of constraints. 

Both methods performed better on (PI) compared to (P2). 

This might well be predicted. For in (PI), all constraint sets are convex, hence convergence is 
guaranteed by Theorem 3.1 and Theorem 2.1, respectively. However, in (P2), the constraints are 
non-convex, thus neither Theorem cannot be evoked. Our results suggest that the cyclic Douglas- 
Rachford as a heuristic. 

We note that there are some difficulties in using the number of iterations as a comparison between 
two methods. 

Each cyclic Douglas-Rachford iteration requires the computation of 27V reflections, and each 
Douglas-Rachford iteration (TV + 1). Even taking this into account, performance of the cyclic 
Douglas-Rachford method was superior to the original Douglas-Rachford method on both (PI) 
and (P2). However, in light of the "no free lunch" theorems of Wolpert and Macready [39], we are 
heedful about asserting dominance of our method. 
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Tabic 1: Results for N ball constraints in R™ with e = 10 3 . The mean (max) from 10 trials are reported 
for the cyclic Douglas-Rachford (cycDR) and Douglas-Rachford (DR) methods. 







Iterations 




Time 


00 




Error 






N 


cycDR 


DR 


cycDR 


DR 


cycDR 


DR 




100 


10 


4.6 


(51 


22.9 (45) 


0.004 


(0.005) 


0.022 (0.041) 


(0) 


7.91e-34 


(1.65e-33) 


100 


20 


3.4 


(4) 


42.4 (113) 


0.006 


(0.007) 


0.071 (0.183) 


(0) 


1.59e-33 


(6.11e-33) 


100 


50 


2.3 




75.3 (241) 


0.008 


(0.011) 


0.288 (0.907) 


2.03e-14 (2.02e-13) 


6.37e-08 


(6.37e-07) 


100 


100 


2.1 


(3) 


97.9 (151) 


0.014 


(0.019) 


0.717 (1.096) 


(0) 


5.51e-33 


(3.85e-32) 


100 


200 


2.0 


(2) 


186.2 (329) 


0.025 


(0.025) 


2.655 (4.656) 


9.68e-15 (9.68e-14) 


2.17e-08 


(2.17e-07) 


100 


500 


2.0 


(2) 


284.2 (372) 


0.059 


(0.060) 


9.968 (12.989) 


(0) 


2.70e-07 


(9.51e-07) 


100 


1000 


2.0 


(2) 


383.0 (507) 


0.118 


(0.119) 


26.656 (35.120) 


(0) 


4.30e-07 


(9.42e-07) 


100 


1100 


2.0 




380.7 (471) 


0.129 


(0.130) 


29.160 (36.001) 


(0) 


8.35e-07 


(1.79e-06) 


100 


1200 


2.0 


(2) 


372.3 (537) 


0.141 


(0.144) 


31.140 (44.886) 


(0) 


8.08e-07 


(1.79e-06) 


100 


1500 


2.0 


(2) 


466.0 (631) 


0.178 


(0.181) 


49.282 (66.533) 


0(0) 


5.38e-05 


(5.34e-04) 


100 


2000 


2.0 


(2) 


529.3 (725) 


0.232 


(0.234) 


74.878 (102.148) 


9.31e-19 (5.29e-18) 


4.79e-06 


(4.00e-05) 


200 


10 


6.3 


(7) 


22.1 (35) 


0.007 


(0.008) 


0.023 (0.036) 


(0) 


1.89e-33 


(6.18e-33) 


200 


20 


4.2 


(h) 

y°) 


23.8 (56) 


0.008 


(0.010) 


0.045 (0.103) 


(0) 


6.61e-33 


(2.55e-32) 


200 


50 


2.8 


("31 


66.4 (144) 


0.012 


(0.013) 


0.283 (0.604) 


(0) 


1.48e-32 


(7.12e-32) 


200 


100 


2.2 


(3) 


81.5 (132) 


0.016 


(0.021) 


0.673 (1.083) 


(0) 


3.20e-32 


(1.03e-31) 


200 


200 


2.0 


(2) 


149.9 (301) 


0.027 


(0.028) 


2.413 (4.801) 


7.84e-16 (7.84e-15) 


5.97e-08 


(5.97e-07) 


200 


500 


2.1 




245.6 (354) 


0.067 


(0.095) 


9.739 (14.055) 


(0) 


2.20e-07 


(8.42e-07) 


200 


1000 


2.0 


(2) 


323.4 (417) 


0.124 


(0.125) 


26.429 (34.023) 


(0) 


4.10e-07 


(9.43e-07) 


200 


1100 


2.1 




358.1 (434) 


0.140 


(0.201) 


32.481 (39.289) 


(0) 


4.06e-07 


(8.92e-07) 


200 


1200 


2.0 


(2) 


337.0 (455) 


0.145 


(0.146) 


33.662 (45.415) 


(0) 


8.51e-07 


(1.63e-06) 


200 


1500 


2.0 


(2) 


379.1 (495) 


0.181 


(0.183) 


48.070 (62.778) 


2.94e-19 (2.94e-18) 


6.70e-07 


(1.36e-06) 


200 


2000 


2.0 


(2) 


422.6 (569) 


0.239 


(0.240) 


74.611 (100.490) 


(0) 


7.28e-05 


(7.22e-04) 


500 


10 


9.1 


fill 


17.0 (37) 


0.012 


(0.014) 


0.023 (0.049) 


(0) 


3.19e-33 


(8.23e-33) 


500 


20 


6.1 


(7) 


16.9 (31) 


0.014 


(0.016) 


0.042 (0.076) 


(0) 


2.35e-32 


(6.76e-32) 


500 


50 


3.0 




66.3 (184) 


0.016 


(0.017) 


0.373 (1.024) 


(0) 


4.55e-32 


(2.23e-31) 


500 


100 


2.6 


(3) 


81.5 (167) 


0.023 


(0.026) 


0.892 (1.804) 


(0) 


2.64e-31 


(1.21e-30) 


500 


200 


2.3 


(3) 


142.5 (251) 


0.037 


(0.046) 


3.068 (5.367) 


(0) 


6.58e-32 


(1.90e-31) 


500 


500 


2.0 


(2) 


267.3 (354) 


0.071 


(0.072) 


15.687 (20.713) 


(0) 


2.40e-07 


(1.22e-06) 


500 


1000 


2.2 


(3) 


318.6 (413) 


0.151 


(0.204) 


42.107 (54.312) 


(0) 


4.33e-07 


(9.15e-07) 


500 


1100 


2.0 


(2) 


338.4 (402) 


0.149 


(0.152) 


49.911 (59.818) 


(0) 


2.45e-07 


(5.58e-07) 


500 


1200 


2.1 


(3) 


356.5 (478) 


0.171 


(0.240) 


57.385 (76.217) 


0(0) 


3.60e-07 


(9.01e-07) 


ouu 


lOUU 


2.0 


(2) 


OA c 7 { Al~\ l 7\ 


0.203 


(0.205) 


7n 070 (so ghq'* 
f U.Z/ z yoZ.ovo) 


(0) 


A 'ICt^ (Y7 


y u . i t e-u / j 


500 


2000 


2.0 


(2) 


358.3 (404) 


0.271 


(0.273) 


97.104 (110.421) 


0(0) 


5.34e-07 


(1.12e-06) 


1000 


10 


15.C 


(16) 


12.4 (26) 


0.024 


(0.026) 


0.023 (0.048) 


2.12e-19 (2.12e-18) 


1.24e-32 


(3.34e-32) 


1000 


20 


8.2 


(9) 


20.4 (71) 


0.024 


(0.027) 


0.069 (0.237) 


0(0) 


3.02e-32 


(6.98e-32) 


1000 


50 


4.3 


(5) 


38.8 (112) 


0.028 


(0.031) 


0.311 (0.884) 


2.67e-19 (2.67e-18) 


1.24e-31 


(5.29e-31) 


1000 


100 


3.3 


(4) 


80.8 (222) 


0.037 


(0.042) 


1.260 (3.436) 


0(0) 


2.15e-31 


(6.84e-31) 


1000 


200 


2.4 


(3) 


138.5 (270) 


0.048 


(0.058) 


4.730 (9.446) 


0(0) 


6.50e-31 


(2.52e-30) 


1000 


500 


2.0 


(2) 


201.3 (313) 


0.085 


(0.086) 


20.356 (31.166) 


3.90e-20 (3.90e-19) 


2.10e-30 


(6.11e-30) 


1000 


1000 


2.0 


(2) 


348.8 (518) 


0.162 


(0.164) 


73.420 (108.493) 


0(0) 


1.36e-06 


(1.20e-05) 


1000 


1100 


2.1 


(3) 


334.4 (550) 


0.183 


(0.260) 


77.174 (126.896) 


0(0) 


1.10e-07 


(7.62e-07) 


1000 


1200 


2.0 


(2) 


353.8 (518) 


0.190 


(0.193) 


89.153 (128.683) 


0(0) 


1.74e-07 


(9.63e-07) 


1000 


1500 


2.1 


(3) 


403.9 (607) 


0.245 


(0.346) 


126.707 (189.011) 


1.33e-19 (1.33e-18) 


3.17e-07 


(8.94e-07) 


1000 


2000 


2.0 


(2) 


487.0 (593) 


0.307 


(0.312) 


239.210 (374.390) 


0(0) 


3.58e-07 


(l.lle-06) 
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Table 2: Results for N ball constraints in M n with e = 10 6 . The mean (max) from 10 trials are reported 
for the cyclic Douglas-Rachford (cycDR) and Douglas-Rachford (DR) methods. 



n 


N 


Iterations 
cycDR 


DR 


Time (s) 
cycDR 


DR 


Error 
cycDR 


DR 




100 


10 


4.7 


(6) 


22.9 (45) 


0.005 


(0.005) 


0.023 (0.044) 





(0) 


7.91e-34 


(1.65e-33) 


100 


20 


3.6 


(5) 


42.4 (113) 


0.006 


(0.008) 


0.077 (0.199) 





(0) 


1.59e-33 


(6.11e-33) 


100 


50 


2.6 


(4) 


77.4 (262) 


0.010 


(0.014) 


0.320 (1.068) 





(0) 


1.24e-32 


(5.96e-32) 


100 


100 


2.1 


(3) 


97.9 (151) 


0.015 


(0.020) 


0.781 (1.195) 





(0) 


5.51e-33 


(3.85e-32) 


100 


200 


2.3 


(3) 


187.1 (329) 


0.029 


(0.038) 


2.909 (5.077) 





(0) 


5.89e-33 


(2.30e-32) 


100 


500 


2.3 


(3) 


329.6 (661) 


0.071 


(0.093) 


12.554 (24.975) 





(0) 


1.81e-32 


(6.37e-32) 


100 


1000 


2.3 


(3) 


427.4 (635) 


0.141 


(0.184) 


32.431 (47.903) 





(0) 


2.21e-32 


(8.10e-32) 


100 


1100 


2.3 


(3) 


467.4 (714) 


0.153 


(0.199) 


38.936 (59.259) 





(0) 


3.92e-32 


(3.17e-31) 


100 


1200 


2.1 


(3) 


451.8 (698) 


0.154 


(0.218) 


41.059 (63.259) 





(0) 


1.12e-31 


(8.08e-31) 


100 


1500 


2.1 


(3) 


507.2 (712) 


0.193 


(0.277) 


58.578 (81.907) 





(0) 


2.66e-31 


(8.15e-31) 


100 


2000 


2.3 


(3) 


627.8 (808) 


0.276 


(0.361) 


96.554 (124.880) 





(0) 


1.50e-31 


(7.53e-31) 


200 


10 


6.3 


(7) 


22.1 (35) 


0.007 


(0.008) 


0.026 (0.040) 





(0) 


1.89e-33 


(6.18e-33) 


200 


20 


4.4 


(5) 


23.8 (56) 


0.009 


(0.010) 


0.050 (0.116) 





(0) 


6.61e-33 


(2.55e-32) 


200 


50 


2.8 


(3) 


66.4 (144) 


0.012 


(0.014) 


0.323 (0.691) 





(0) 


1.48e-32 


(7.12e-32) 


200 


100 


2.4 


(3) 


81.5 (132) 


0.018 


(0.022) 


0.772 (1.242) 


o 


CO) 


3.20e-32 


(1.03e-31) 


200 


200 


2.1 


(3) 


152.5 (301) 


0.030 


(0.040) 


2.825 (5.547) 





(0) 


3.04e-32 


(1.63e-31) 


200 


500 


2.5 


(3) 


263.8 (435) 


0.081 


(0.098) 


12.074 (19.831) 





(0) 


4.32e-32 


(2.69e-31) 


200 


1000 


2.1 


(3) 


427.9 (703) 


0.135 


(0.192) 


40.025 (65.394) 





(0) 


6.64e-32 


(2.66e-31) 


200 


1100 


2.2 


(3) 


426.0 (545) 


0.153 


(0.209) 


44.161 (56.724) 





(0) 


5.92e-32 


(1.86e-31) 


200 


1200 


2.2 


(3) 


442.9 (633) 


0.166 


(0.225) 


50.678 (72.862) 





(0) 


5.98e-32 


(2.81e-31) 


200 


1500 


2.1 


(3) 


470.1 (882) 


0.196 


(0.279) 


69.261 (128.978) 


1. 


00e-25 (1.00e-24) 


1.71e-31 


(6.88e-31) 


200 


2000 


2.0 


(2) 


578.4 (894) 


0.248 


(0.252) 


117.575 (179.883) 





(0) 


4.82e-32 


(1.04e-31) 


500 


10 


9.1 


(11) 


17.0 (37) 


0.012 


(0.015) 


0.028 (0.060) 





(0) 


3.19e-33 


(8.23e-33) 


500 


20 


6.1 


(7) 


16.9 (31) 


0.015 


(0.017) 


0.052 (0.093) 





(0) 


2.35e-32 


(6.76e-32) 


500 


50 


3.1 


(4) 


66.3 (184) 


0.017 


(0.019) 


0.467 (1.285) 





(0) 


4.55e-32 


(2.23e-31) 


500 


100 


2.6 


(3) 


81.5 (167) 


0.024 


(0.027) 


1.132 (2.287) 





(0) 


2.64e-31 


(1.21e-30) 


500 


200 


2.7 


(4) 


142.5 (251) 


0.043 


(0.060) 


3.979 (6.824) 





(0) 


6.58e-32 


(1.90e-31) 


500 


500 


2.1 


(3) 


277.5 (399) 


0.078 


(0.108) 


20.528 (29.207) 





(0) 


4.06e-31 


(2.22e-30) 


500 


1000 


2.3 


(3) 


358.3 (540) 


0.162 


(0.210) 


59.290 (88.063) 





(0) 


8.30e-32 


(3.91e-31) 


500 


1100 


2.1 


(3) 


372.7 (458) 


0.163 


(0.231) 


67.065 (83.951) 





(0) 


6.41e-32 


(3.21e-31) 


500 


1200 


2.2 


(3) 


416.4 (604) 


0.184 


(0.246) 


82.461 (119.456) 





(0) 


4.81e-32 


(2.22e-31) 


500 


1500 


2.1 


(3) 


461.7 (691) 


0.220 


(0.313) 


114.836 (175.009) 





(0) 


2.28e-31 


(1.36e-30) 


500 


2000 


2.0 


(2) 


483.9 (785) 


0.278 


(0.283) 


159.287 (259.033) 





(0) 


6.06e-31 


(2.92e-30) 


1000 


10 


15.1 


(17) 


12.4 (26) 


0.024 


(0.027) 


0.030 (0.063) 





(0) 


1.24e-32 


(3.34e-32) 


1000 


20 


8.2 


(9) 


20.4 (71) 


0.025 


(0.027) 


0.095 (0.330) 





(0) 


3.02e-32 


(6.98e-32) 


1000 


50 


4.5 


(6) 


38.8 (112) 


0.029 


(0.035) 


0.434 (1.249) 





(0) 


1.24e-31 


(5.29e-31) 


1000 


100 


3.3 


(4) 


80.8 (222) 


0.038 


(0.043) 


1.761 (4.730) 





(0) 


2.15e-31 


(6.84e-31) 


1000 


200 


2.5 


(3) 


138.5 (270) 


0.051 


(0.059) 


6.224 (12.089) 





(0) 


6.50e-31 


(2.52e-30) 


1000 


500 


2.3 


(3) 


201.3 (313) 


0.099 


(0.125) 


26.108 (40.534) 





(0) 


2.10e-30 


(6.11e-30) 


1000 


1000 


2.1 


(3) 


388.7 (905) 


0.174 


(0.241) 


103.839 (243.085) 





(0) 


2.17e-30 


(1.79e-29) 


1000 


1100 


2.3 


(3) 


354.4 (660) 


0.205 


(0.264) 


120.706 (220.612) 





(0) 


2.26e-30 


(9.82e-30) 


1000 


1200 


2.3 


(3) 


376.3 (620) 


0.223 


(0.288) 


161.133 (260.857) 





(0) 


1.61e-30 


(1.26e-29) 


1000 


1500 


2.2 


(3) 


526.0 (1000) 


0.265 


(0.358) 


276.095 (541.502) 


2. 


68e-22 (2.68e-21) 


1.08e-09 


(5.98e-09) 


1000 


2000 


2.1 


(3) 


595.0 (894) 


0.332 


(0.469) 


427.933 (646.182) 





(0) 


4.48e-31 


(1.97e-30) 
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Table 3: Results for N sphere constraints in R" with e = 10 3 . The mean (max) from 10 trials are 
reported for the cyclic Douglas-Rachford (cycDR) and Douglas-Rachford (DR) methods. 



n 


N 


Iterations 
cycDR 


DR 


100 


10 


16.8 (17) 


219.1 (327) 


100 


20 


9.0 (9) 


247.8 (314) 


100 


50 


5.0 (5) 


375.1 (481) 


100 


100 


3.0 (3) 


471.6 (806) 


100 


200 


2.0 (2) 


747.7 (1000) 


100 


500 


2.0 (2) 


1000.0 (1000) 


100 


1000 


2.0 (2) 


1000.0 (1000) 


100 


1100 


2.0 (2) 


1000.0 (1000) 


100 


1200 


2.0 (2) 


1000.0 (1000) 


100 


1500 


2.0 (2) 


1000.0 (1000) 


100 


2000 


2.0 (2) 


1000.0 (1000) 


200 


10 


23.0 (23) 


123.1 (222) 


200 


20 


12.8 (13) 


115.2 (171) 


200 


50 


6.0 (6) 


110.6 (124) 


200 


100 


4.0 (4) 


120.1 (128) 


200 


200 


3.0 (3) 


134.9 (139) 


200 


500 


2.0 (2) 


156.4 (161) 


200 


1000 


2.0 (2) 


175.6 (182) 


200 


1100 


2.0 (2) 


179.5 (191) 


200 


1200 


2.0 (2) 


179.0 (184) 


200 


1500 


2.0 (2) 


190.0 (200) 


200 


2000 


2.0 (2) 


230.3 (295) 


500 


10 


35.3 (36) 


51.6 (67) 


500 


20 


19.1 (20) 


72.3 (85) 


500 


50 


9.0 (9) 


96.8 (107) 


500 


100 


5.0 (5) 


120.5 (127) 


500 


200 


3.0 (3) 


143.0 (148) 


500 


500 


2.0 (2) 


171.3 (176) 


500 


1000 


2.0 (2) 


195.1 (197) 


500 


1100 


2.0 (2) 


198.1 (202) 


500 


1200 


2.0 (2) 


1QQ 8 ("9041 

133.0 1 1 


500 


1500 


2.0 (2) 


208.5 (213) 


500 


2000 


2.0 (2) 


217.8 (221) 


1000 


10 


49.2 (50) 


9.1 (29) 


1000 


20 


27.0 (27) 


30.0 (66) 


1000 


50 


12.0 (12) 


73.1 (86) 


1000 


100 


7.0 (7) 


103.7 (113) 


1000 


200 


4.0 (4) 


136.8 (143) 


1000 


500 


3.0 (3) 


178.9 (182) 


1000 


1000 


2.0 (2) 


211.7 (215) 


1000 


1100 


2.0 (2) 


215.3 (221) 


1000 


1200 


2.0 (2) 


218.7 (220) 


1000 


1500 


2.0 (2) 


228.6 (232) 


1000 


2000 


2.0 (2) 


242.3 (245) 



Time (s) 
cycDR 



DR 



Error 
cycDR 



DR 



0.021 
0.022 
0.031 
0.037 
0.050 
0.127 
0.253 
0.278 
0.304 
0.380 
0.504 

0.030 
0.033 
0.038 
0.051 
0.077 
0.130 
0.262 
0.286 
0.309 
0.394 
0.522 

0.051 
0.055 
0.064 
0.070 
0.085 
0.145 
0.295 
0.327 
0.359 
0.445 
0.590 

0.083 
0.092 
0.100 
0.117 
0.133 
0.258 
0.343 
0.379 
0.411 
0.518 
0.681 



0.021) 
0.022) 
0.031) 
0.037) 
0.050) 
0.128) 
0.255) 
0.281) 
0.306) 
0.386) 
0.511) 

0.030) 
0.034) 
0.038) 
0.052) 
0.078) 
0.131) 
0.264) 
0.290) 
0.316) 
0.400) 
0.525) 

0.052) 
0.057) 
0.064) 
0.071) 
0.085) 
0.146) 
0.296) 
0.329) 
0.362) 
0.451) 
0.598) 

0.085) 
0.092) 
0.100) 
0.117) 
0.134) 
0.260) 
0.344) 
0.383) 
0.414) 
0.524) 
0.684) 



0.272 (0.421) 
0.669 (0.873) 
2.559 (3.307) 
6.185 (10.904) 
19.932 (26.634) 
64.046 (65.562) 
130.475 (138.540) 
143.022 (149.895) 
156.653 (158.918) 
197.801 (210.661) 
261.535 (267.483) 

0.183 (0.334) 
0.329 (0.507) 
0.790 (0.874) 

I. 726 (1.825) 
3.749 (4.088) 

II. 106 (11.715) 
26.888 (30.935) 
31.161 (33.273) 
31.547 (35.242) 
43.207 (47.057) 
72.760 (94.718) 

0.093 (0.121) 
0.254 (0.300) 
0.888 (0.991) 
2.271 (2.475) 
5.579 (6.072) 
17.719 (21.106) 
47.771 (51.291) 
50.934 (54.122) 
56.155 (60.472) 
73.848 (78.355) 
100.538 (111.140) 

0.023 (0.072) 
0.127 (0.276) 
0.779 (0.946) 
2.248 (2.513) 
8.869 (10.028) 
31.706 (34.394) 
73.182 (78.028) 
84.584 (92.095) 
94.408 (99.951) 
124.265 (132.683) 
176.575 (191.354) 



4 


46e- 


13 


(7 


24e-13) 


8 


29e-06 


(1 


06e-05) 


5 


94e- 


14 


(1 


12e-13) 


1 


54e-05 


(1 


70e-05) 


6 


59e- 


18 


(1 


00e-17) 


2 


86e-05 


(3 


29e-05) 


1 


30e- 


20 


(2 


62e-20) 


4 


30e-05 


(4 


98e-05) 


3 


60e- 


26 


(4 


50e-26) 


5 


66e-05 


(6 


12e-05) 


2 


56e- 


26 


(5 


32e-26) 


1 


18e-04 


(1 


40e-04) 


3 


87e- 


26 


(8 


28e-26) 


2 


43e-04 


(2 


70e-04) 


5 


28e- 


26 


(8 


95e-26) 


2 


53e-04 


(2 


95e-04) 


7 


16e- 


26 


(1 


65e-25) 


3 


12e-04 


(3 


74e-04) 


1 


02e- 


25 


(2 


27e-25) 


3 


50e-04 


(3 


84e-04) 


9 


91e- 


26 


(2 


42e-25) 


4 


82e-04 


(6 


04e-04) 


2 


50e- 


13 


(7 


46e-13) 


6 


33e-06 


(8 


72e-06) 


1 


48e- 


14 


(4 


39e-14) 


1 


05e-05 


(1 


46e-05) 


2 


56e- 


16 


(4 


47e-16) 


1 


42e-05 


(2 


09e-05) 


2 


49e- 


20 


(3 


71e-20) 


1 


70e-05 


(2 


21e-05) 


2 


88e- 


26 


(6 


69e-26) 


2 


31e-05 


(2 


98e-05) 


8 


53e- 


26 


(1 


71e-25) 


4 


37e-05 


(5 


16e-05) 


1 


53e- 


25 


(3 


33e-25) 


7 


27e-05 


(8 


71e-05) 


1 


71e- 


25 


(2 


77e-25) 


7 


97e-05 


(9 


82e-05) 


2 


02e- 


25 


(4 


76e-25) 


7 


86e-05 


(8 


59e-05) 


2 


29e- 


25 


(3 


91e-25) 


9 


97e-05 


(1 


15e-04) 


3 


96e- 


25 


(7 


53e-25) 


1 


34e-04 


(1 


58e-04) 


4 


81e- 


14 


(1 


13e-13) 


1 


46e-06 


(2 


86e-06) 


8 


32e- 


15 


(1 


21e-14) 


2 


02e-06 


(3 


29e-06) 


1 


82e- 


16 


(2 


72e-16) 


2 


03e-06 


(2 


36e-06) 


1 


21e- 


17 


(1 


75e-17) 


2 


39e-06 


(2 


98e-06) 


4 


29e- 


20 


(5 


80e-20) 


2 


84e-06 


(3 


79e-06) 


3 


30e- 


25 


(8 


09e-25) 


4 


14e-06 


(4 


50e-06) 


8 


61e- 


25 


(1 


37e-24) 


6 


18e-06 


(6 


64e-06) 


1 


02e- 


24 


(2 


28e-24) 


6 


93e-06 


(8 


30e-06) 


1 


Ole- 


24 


(2 


17e-24) 


6 


69e-06 


(7 


56e-06) 


1 


34e- 


24 


(2 


66e-24) 


7 


96e-06 


(8 


62e-06) 


1 


61e- 


24 


(3 


00e-24) 


1 


00e-05 


(1 


09e-05) 


1 


32e- 


14 


(2 


44e-14) 


3 


15e-07 


(7 


lle-07) 


1 


96e- 


15 


(3 


lle-15) 


4 


88e-07 


(7 


90e-07) 


1 


85e- 


16 


(2 


37e-16) 


4 


98e-07 


(6 


57e-07) 


4 


22e- 


18 


(5 


49e-18) 


5 


51e-07 


(7 


17e-07) 


8 


89e- 


20 


(1 


le-19) 


6 


28e-07 


(7 


86e-07) 


2 


17e- 


24 


(5 


88e-24) 


7 


86e-07 


(9 


48e-07) 


2 


16e- 


24 


(3 


71e-24) 


1 


04e-06 


(1 


15e-06) 


4 


Ole- 


24 


(9 


45e-24) 


1 


07e-06 


(1 


21e-06) 


3 


91e- 


24 


(8 


19e-24) 


1 


14e-06 


(1 


27e-06) 


5 


73e- 


24 


(1 


58e-23) 


1 


29e-06 


(1 


48e-06) 


6 


06e- 


24 


(1 


5e-23) 


1 


53e-06 


(1 


67e-06) 
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Table 4: Results for N sphere constraints in R™ with e = 10 6 . The mean (max) from 10 trials are 
reported for the cyclic Douglas-Rachford (cycDR) and Douglas-Rachford (DR) methods. 



n 


N 


Iterations 
cycDR 


DR 






Time (s) 
cycDR 


DR 


Error 
cycDR 






DR 






100 


10 


27.4 (28) 


1000 





(1000) 


0.035 


(0.036) 


1.302 (1.419) 


1.21e- 


18 


(2.25e 


18) 


9 


10e-08 


(2 


16e-07) 


100 


20 


14.1 (15) 


1000 





(1000) 


0.036 


(0.038) 


2.463 (2.750) 


1.21e- 


19 


(2.65e 


19) 


1 


26e-06 


(1 


78e-06) 


100 


50 


7.0 (7) 


1000 





(1000) 


0.044 


(0.045) 


6.760 (7.052) 


1.02e- 


23 


(1.81e 


23) 


8 


51e-06 


(1 


07e-05) 


100 


100 


4.0 (4) 


1000 





(1000) 


0.052 


0.052) 


13.823 (14.145) 


2.02e- 


26 


(3.73e 


26) 


2 


17e-05 


(3 


00e-05) 


100 


200 


3.0 (3) 


1000 





(1000) 


0.078 


(0.078) 


25.239 (27.594) 


8.97e- 


27 


(1.69e 


26) 


4 


39e-05 


(5 


93e-05) 


100 


500 


2.0 (2) 


1000 





(1000) 


0.131 


(0.132) 


66.159 (68.491) 


2.56e- 


26 


(5.32e 


26) 


1 


18e-04 


(1 


40e-04) 


100 


1000 


2.0 (2) 


1000 





(1000) 


0.262 


(0.263) 


131.165 (139.166) 


3.87e- 


26 


(8.28e 


26) 


2 


43e-04 


(2 


70e-04) 


100 


1100 


2.0 (2) 


1000 





(1000) 


0.290 


(0.293) 


149.386 (154.285) 


5.28e- 


26 


(8.95e 


26) 


2 


53e-04 


(2 


95e-04) 


100 


1200 


2.0 (2) 


1000 





(1000) 


0.317 


(0.322) 


162.476 (171.252) 


7.16e- 


26 


(1.65e 


25) 


3 


12e-04 


(3 


74e-04) 


100 


1500 


2.0 (2) 


1000 





(1000) 


0.395 


(0.399) 


205.210 (214.347) 


1.02e- 


25 


(2.27e 


25) 


3 


50e-04 


(3 


84e-04) 


100 


2000 


2.0 (2) 


1000 





(1000) 


0.524 


(0.527) 


284.740 (295.621) 


9.91e- 


26 


(2.42e 


25) 


4 


82e-04 


(6 


04e-04) 


200 


10 


37.8 (39) 


1000 





(1000) 


0.051 


(0.053) 


1.787 (1.801) 


5.36e- 


19 


(9.86e 


19) 


9 


14e-08 


(1 


73e-07) 


200 


20 


20.0 (20) 


1000 





(1000) 


0.053 


(0.054) 


3.422 (3.452) 


2.01e- 


20 


(3.49e 


20) 


9 


56e-07 


(1 


46e-06) 


200 


50 


9.0 (9) 


1000 





(1000) 


0.059 


(0.060) 


8.384 (8.615) 


1.53e- 


22 


(3.08e 


22) 


4 


52e-06 


(6 


27e-06) 


200 


100 


5.0 (5) 


1000 





(1000) 


0.067 


0.067) 


15.429 (17.471) 


1.61e- 


24 


(2.45e 


24) 


8 


05e-06 


(1 


09e-05) 


200 


200 


3.0 (3) 


1000 





(1000) 


0.080 


(0.080) 


31.967 (33.857) 


2.88e- 


26 


(6.69e 


26) 


1 


39e-05 


(1 


8e-05) 


200 


500 


2.0 (2) 


1000 





(1000) 


0.135 


(0.135) 


81.272 (85.423) 


8.53e- 


26 


(1.71e 


25) 


3 


07e-05 


(3 


64e-05) 


200 


1000 


2.0 (2) 


1000 





(1000) 


0.272 


(0.273) 


166.615 (177.342) 


1.53e- 


25 


(3.33e 


25) 


5 


49e-05 


(6 


55e-05) 


200 


1100 


2.0 (2) 


1000 





(1000) 


0.297 


(0.299) 


168.501 (184.769) 


1.71e- 


25 


(2.77e 


25) 


6 


05e-05 


(7 


36e-05) 


200 


1200 


2.0 (2) 


1000 





(1000) 


0.320 


(0.323) 


195.997 (204.751) 


2.02e- 


25 


(4.76e 


25) 


6 


03e-05 


(6 


58e-05) 


200 


1500 


2.0 (2) 


1000 





(1000) 


0.411 


(0.416) 


250.555 (257.482) 


2.29e- 


25 


(3.91e 


25) 


7 


77e-05 


(9 


00e-05) 


200 


2000 


2.0 (2) 


1000 





(1000) 


0.540 


(0.543) 


333.273 (340.514) 


3.96e- 


25 


(7.53e 


25) 


1 


06e-04 


(1 


29e-04) 


500 


10 


58.0 (59) 


1000 





(1000) 


0.085 


(0.087) 


2.135 (2.220) 


1.46e- 


19 


(3.30e 


19) 


7 


50e-08 


(1 


05e-07) 


500 


20 


30.8 (31) 


1000 





(1000) 


0.091 


(0.091) 


3.658 (3.691) 


1.04e- 


20 


(2.56e 


20) 


4 


45e-07 


(6 


81e-07) 


500 


50 


13.1 (14) 


1000 





(1000) 


0.095 


(0.102) 


9.321 (10.090) 


8.52e- 


22 


(1.38e 


21) 


1 


05e-06 


(1 


21e-06) 


500 


100 


7.8 (8) 


1000 





(1000) 


0.114 


(0.117) 


18.124 (19.334) 


8.23e- 


24 


(4.40e 


23) 


1 


65e-06 


(2 


04e-06) 


500 


200 


5.0 (5) 


1000 





(1000) 


0.147 


(0.147) 


41.555 (45.159) 


1.60e- 


25 


(2.81e 


25) 


2 


25e-06 


(2 


95e-06) 


500 


500 


3.0 (3) 


1000 





(1000) 


0.224 


(0.225) 


118.550 (125.955) 


3.31e- 


25 


(8.15e 


25) 


3 


60e-06 


(3 


91e-06) 


500 


1000 


2.0 (2) 


1000 





(1000) 


0.305 


(0.306) 


256.931 (276.971) 


8.61e- 


25 


(1.37e 


24) 


5 


57e-06 


(5 


97e-06) 


500 


1100 


2.0 (2) 


1000 





(1000) 


0.336 


(0.338) 


279.305 (295.475) 


1.02e- 


24 


(2.28e 


24) 


6 


26e-06 


(7 


46e-06) 


500 


1200 


2.0 (2) 


1000 





(1000) 


0.369 


(0.371) 
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6 Conclusion 



Two new projection algorithms, the cyclic Douglas-Rachford and averaged Douglas-Rachford iteration 
schemes, were introduced and studied. Applied to N-set convex feasibility problems in Hilbert space, 
both weakly converge to point whose projections onto each of the iV-set coincide. While the cyclic 
Douglas-Rachford is sequential, each iteration of the averaged Douglas-Rachford can be parallelized. 

Numerical experiments suggest that that the cyclic Douglas-Rachford scheme outperforms the clas- 
sical Douglas-Rachford scheme, which suffers as a result of the product formulation. An advantage of 
our schemes is that they can be used in the original space, without recourse to this formulation. For 
inconsistent 2-set problems, there is evidence to suggest that the two set cyclic Douglas-Rachford scheme 
yields best approximation pairs. 

HTML versions of the interactive Cinderella applets are available at: 

1. http : / / carma.newcastle . edu.au/tam/ cycdr/21ines .html 

2. http : / / carma . newcastle . edu . au/tam/cycdr/ circleline . html 

3. http : / / carma . newcastle . edu . au/tam/ cycdr/2circles . html 

4. http : / / carma . newcastle . edu . au/tam/cycdr/ circlepoint . html 
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